Terrain  Display  Alternatives: 
Assessment  of  Information  Density 
and  Alerting  Strategies 


•  DOT/FAA/AAR- 100-98-1 

DOT-VNTSC-FAA-98-2 

Office  of  Aviation  Research 
Washington,  DC  20591 


U.S.  Department  of  Transportation 
Research  and  Special  Programs  Administration 
John  A.  Voipe  National  Transportation  Systems  Center 
Cambridge,  MA  02142 


Final  Report 
April  1998 


This  document  is  available  to  the  public 
through  the  National  Technical  Information 
Service,  Springfield,  Virginia  22161 


U.S.  Department  of  Transportation 

Federal  Aviation  Administration 


19980807  062 


NOTICE 


This  document  is  disseminated  under  the  sponsorship  of  the 
Department  of  Transportation  in  the  interest  of  information 
exchange.  The  United  States  Government  assumes  no  liability  for 
its  contents  or  use  thereof. 


NOTICE 

The  United  States  Government  does  not  endorse  products  or 
manufacturers.  Trade  or  manufacturers'  names  appear  herein 
solely  because  they  are  considered  essential  to  the  objective  of  this 
report. 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
0MB  No.  V704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  pej  response, 
including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments 
regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  infprmation,  including 
suggestions  for  reducing  this  burden,  to  Washington  Headguarters  Services,  Directorate  for  Information 
Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the 
Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188),  Washington,  DC  20503. 


1.  AGENCY  USE  ONLY  (Leave  blank) 


REPORT  DATE 

April  1998 


3.  REPORT  TYPE  AND  DATES  COVERED 

Final  Report 

June  1996  -  September  1997 


4.  TITLE  AND  SUBTITLE 

Terrain  Display  Alternatives:  Assessment  of  Information 
Density  and  Alerting  Strategies  _ ^ _ 


5.  FUNDING  NUMBERS 


A8123/FA8E2 


6.  AUTHOR(S) 

Melissa  Bud,*  Mary  Stearns,  Peter  Mengert* _ 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS (ES) 

U.S.  Department  of  Transportation 

Research  and  Special  Programs  Administration 

John  A.  Volpe  National  Transportation  Systems  Center 

Cambridge,  MA  02142-1093  _ 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 
U.S.  Department  of  Transportation 
Federal  Aviation  Administration 
Research  and  Development  Service 
800  Independence  Ave.  S.W. 

Washington  DC  20591  _ 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 

DOT-VNTSC-FAA-98-2 


10.  SPONSORING/MONITORING 
AGENCY  REPORT  NUMBER 

DOT/FAA/AAR -100-98-1 


12a.  DISTRIBUTION/AVAILABILITY  STATEMENT  12b.  DISTRIBUTION  CODE 

This  document  is  available  to  the  public  through  the  National 
Technical  Information  Service,  Springfield,  VA  22161 


13.  ABSTRACT  (Maximum  200  words) 

Current  technology  makes  it  possible  to  display  navigation  and  terrain  information  on  electronic 
screens  in  the  cockpit.  The  conventions  used  for  position  and  terrain  information  must  be  clearly 
presented  so  pilots  can  maintain  their  positional  awareness  and  avoid  collision  with  terrain.  There 
are  very  few  recommendations  or  guidelines  available  for  the  design  and  evaluation  of  electronic 
displays . 

The  Volpe  National  Transportation  Systems  Center  (Volpe  Center) ,  in  support  of  the  Federal  Aviation 
Administration’s  (FAA's)  Office  of  the  Chief  Scientific  and  Technical  Advisor  for  Human  Factors, 

AAR- 100,  conducted  a  series  of  experiments  to  explore  human  factors  issues  in  depicting  terrain  on 
electronic  displays.  Experiments  examined  instrument -rated  General  Aviation  (GA)  pilots’  ability  to 
interpret  terrain  shown  on  electronic  plan  view  displays  in  a  flight  simulator.  The  four  experiments 
examined  the  following  electronic  display  alternatives :  the  utility  of  presenting  nonthreatening 
terrain  during  approaches;  effectiveness  of  showing  only  terrain  features  for  terrain  avoidance; 
preferences  for  information  density;  and  the  utility  of  terrain  displays  paired  with  visual  terrain 
alerts.  Results  suggest  display  designs  should  incorporate  alerts  that  identify  danger  and  reorient 
pilots.  Electronic  displays  with  terrain  that  incorporate  these  recommendations  should  diminish  the 
number  of  accidents  resulting  from  a  loss  of  positional  awareness,  such  as 
controlled- flight- into- terrain  (CFIT)  accidents. 


14.  SUBJECT  TERMS 

avionics,  controlled- flight- into  terrain  (CFIT)  accidents,  cross 
track  error  (XTE),  electronic  displays,  terrain  display 


15.  NUMBER  OF  PAGES 
54 


16.  PRICE  CODE 


17.  SECURITY 
CLASSIFICATION 
OF  REPORT 
Unclassified 


18.  SECURITY 
CLASSIFICATION 

OF  THIS  PAGE 

Unclassified 


19.  SECURITY 
CLASSIFICATION 
OF  ABSTRACT 
Unclassified 


20.  LIMITATION  OF 
ABSTRACT 


NSN  7540-01-280-5500 

STIC  QUAimr  INSPECTED  I 


Standard  Form  298  (Rev.  2-89) 
Prescribed  by  ANSI  Std.  239- i8 
298-102 


PREFACE 


Electronic  display  technology  is  a  recent  development  offering  many  possibilities  for  visual 
representations  of  terrain  information.  Electronic  displays  of  terrain  information  have  the 
potential  to  increase  pilot  positional  awareness  and  to  aid  in  terrain  avoidance.  This  report 
describes  a  series  of  four  experiments  conducted  to  identify  human  factors  issues  associated  with 
presenting  terrain  information  on  electronic  displays.  The  experimental  results  provide 
information  to  pilots,  aviation  regulators,  and  display  designers  to  help  them  ensure  that  this  new 
technology  is  used  effectively  to  improve  aviation  safety. 

The  experimental  research  was  performed  in  the  Cockpit  Human  Factors  laboratory  located  at  the 
Volpe  National  Transportation  Systems  Center  (Volpe  Center).  In  the  experiments,  researchers 
examined  electronic  display  presentations,  positional  awareness,  terrain  alerts,  and  terrain 
avoidance. 
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EXECUTIVE  SUMMARY 


Current  technology  makes  navigation  and  terrain  information  available  on  electronic  display 
screens  in  the  cockpit.  This  information  must  be  presented  clearly  for  pilots  to  maintain 
positional  awareness  and  to  avoid  collision  with  terrain.  However,  there  are  few 
recommendations  or  guidelines  as  to  how  electronic  displays  of  information  should  be  designed 
or  evaluated. 

Electronic  displays  present  enhanced  information  to  the  pilot  which  may  help  to  reduce  the 
accident  rate  in  General  Aviation  (GA)  flight.  For  example,  in  the  United  States  from  1983  to 
1994,  Controlled-Flight-Into-Terrain  (CFIT)  incidents  accounted  for  32%  of  the  GA  accidents  in 
instrument  weather  conditions.'  Enhanced  real-time  positional  information  on  electronic  displays 
might  help  to  prevent  collisions  with  terrain. 

The  Volpe  National  Transportation  Systems  Center  (Volpe  Center),  under  the  sponsorship  of  the 
Federal  Aviation  Administration’s  (FAA’s)  Office  of  the  Chief  Scientific  and  Technical  Advisor 
for  Human  Factors,  AAR- 100,  conducted  a  series  of  interrelated  experiments  to  explore  the 
human  factors  issues  in  depicting  terrain  on  electronic  displays.  The  series  of  experiments 
examined  instrument-rated  GA  pilots’  ability  to  interpret  terrain  depicted  on  electronic  plan  view 
displays  using  a  flight  simulator. 

Because  the  resolution  of  electronic  display  screens  causes  a  disproportionate  relationship 
between  the  airplane  symbol  and  the  terrain  features,  the  initial  experiment  examined  the  utility 
of  displaying  nonthreatening  terrain  to  pilots  during  approaches.  Participants  flew  the 
approaches  accurately  and  did  not  veer  off  course  despite  the  disproportionate  size  of  the  airplane 
symbol  relative  to  terrain  features. 

Terrain  elevation  information,  presented  on  a  supplemental  plan  view  display,  might  prove  useful 
to  detect  and  avoid  dangerous  terrain.  In  the  second  experiment,  pilots  failed  to  make  use  of  the 
terrain  information  presented  on  an  electronic  display.  When  they  were  presented  with  a  display 
showing  only  terrain  features,  pilots  showed  heightened  awareness  to  terrain  but  not  enough  to 
take  corrective  action.  When  the  display  presented  showed  navigation  and  terrain  information, 
pilots  failed  to  detect  the  dangerous  terrain. 

The  third  experiment  measured  pilots’  preferences  for  information  density  on  plan  view 
electronic  displays.  Despite  findings  that  pilots  may  better  recognize  potential  obstacles  using 
displays  with  a  lower  density  of  information,  participants  preferred  the  displays  with  higher 
content  levels. 

Visual  alerts  may  enhance  the  interpretability  of  electronic  plan  view  displays.  In  the  final 
experiment,  pilots  used  an  electronic  display  paired  with  a  visual  terrain  alert.  Using  alternate 


*  Bud,  M.,  Mengert,  P.,  Ransom,  S.,  and  Steams,  M.D.  General  Aviation  Accidents,  1983-1994:  Identification  of 
Factors  Related  to  Controlled-Flight-lnto-Terrain  (CFIT)  Accidents.  Final  Report  DOT-VNTSC-FAA-97-8.  May 
1997. 


map  formats,  the  pilots  were  measured  to  see  how  well  they  interpreted  these  displays  to  obtain 
terrain  elevation  information  when  there  was  a  potential  problem.  The  map  formats  varied  in 
terms  of  density  of  terrain  and  navigation  information.  The  pilots  were  able  to  interpret  the 
displays  with  equal  facility  regardless  of  the  map  format. 

This  series  of  experiments  addressed  several  human  factors  issues  associated  with  presenting 
terrain  information  on  electronic  displays.  The  summation  of  these  points  is  that  terrain 
information  on  electronic  displays  helps  pilots  to  avoid  terrain  if  combined  with  a  terrain  alerting 
system,  regardless  of  the  information  density  of  the  map.  These  results  suggest  that  designs  for 
display  formats  should  incorporate  alerts  to  make  pilots  aware  of  danger  and  reorient  them 
quickly.  Display  formats  that  incorporate  these  recommendations  should  contribute  to  a 
reduction  in  the  number  of  accidents  which  result  from  a  loss  of  positional  awareness,  such  as 
CFIT  accidents.  Further  research  into  information  presentation  options  and  systems  and  the 
benefit  of  training  is  needed. 


1.  INTRODUCTION 


Current  technology  consolidates  navigation  and  safety  information,  such  as  weather  and  traffic, 
on  electronic  display  screens  in  the  cockpit.  The  information  displayed  must  be  clear  to  allow 
pilots  to  maintain  positional  awareness  and  avoid  collision  with  terrain.  However,  there  are  few 
recommendations  or  guidelines  available  on  how  to  design  or  evaluate  the  acceptable  display 
configurations. 

Electronic  displays  also  incorporate  additional  information  that,  until  recently,  could  not  be 
presented.  This  increased  information  available  to  the  pilot  may  contribute  to  reducing  the 
General  Aviation  (GA)  accident  rate.  Because  research  has  shown  the  unchanged  incidence  of 
Controlled-Flight-Into-Terrain  (CFIT)  accidents  in  GA  operations  (32%  in  instrument  conditions 
from  1983  to  1994^),  the  Volpe  Center  and  the  Federal  Aviation  Administration’s  (FAA’s) 
Office  of  the  Chief  Scientific  and  Technical  Advisor  for  Human  Factors,  AAR- 100,  conducted 
experiments  to  examine  the  potential  value  of,  and  issues  in,  consolidating  and  depicting 
additional  information  using  electronic  displays. 


2.  PURPOSE 

Technological  development  in  avionics,  specifically  in  electronic  display  screens,  computer 
technology,  and  memory  processors,  is  expanding  rapidly  and  the  human  factors  issues  that  are 
inherent  in  these  technologies  warrant  examination.  The  Volpe  Center,  under  the  sponsorship  of 
the  FAA’s  AAR-100,  conducted  a  series  of  experiments  to  examine  how  quickly  and  accurately 
pilots  are  able  to  interpret  alternative  presentations  of  information  on  electronic  display  screens. 


3.  PROCEDURES 


3.1  PARTICIPANTS 

Participants  in  the  experiments  were  general  aviation  (GA)  pilots  with  multi-engine  and 
instmment  ratings.  The  pilots  were  recruited  from  the  Subject  Database  maintained  at  the  Center 
for  Human  Factors  Research  at  the  Volpe  National  Transportation  Systems  Center  (Volpe 
Center)  and  from  fliers  posted  at  local  airports.^ 


^  Bud,  M.,  Mengert,  P.,  Ransom,  S.,  and  Steams,  M.D.  General  Aviation  Accidents,  1983-1994:  Identification  of 
Factors  Related  to  Controlled-Flight-Into-Terrain  (CFIT)  Accidents.  Final  Report  DOT-VNTSC-FAA-97-1.  May 
1997. 

^  Due  to  the  time  (a  minimum  of  four  hours  per  participant)  and  cost  requirements  with  pilot  participants,  sample 
sizes  were  kept  small.  Although  these  sample  sizes  were  not  large  enough  to  provide  significant  statistical  power, 
they  were  considered  adequate  for  identifying  trends  in  the  data. 


3.2  APPARATUS 


The  Center  for  Human  Factors  Research  at  the  Volpe  Center  maintains  a  Frasca  242  light  twin 
engine  instrument  flight  simulator  which  was  used  for  these  experiments.  This  simulator  is  a 
fixed-base  training  device  reconfigured  and  instrumented  for  data  collection.  It  did  not  have  an 
outside  the  window  (OTW)  display  at  the  time  of  these  experiments. 

The  flight  dynamics  of  the  simulator  were  designed  to  resemble  a  Piper  Aztec.  (Appendix  A-1 
shows  the  Frasca  242  simulator.)  There  is  a  picture  window  in  the  wall  behind  the  simulator 
which  permitted  the  researchers  to  observe  the  pilot  during  testing.  Researchers  also  observed 
the  pilot  on  a  television  monitor  located  outside  the  simulator  room  using  a  closed-circuit 
television.  During  the  experiments,  researchers  communicated  with  the  pilots  using  a  standard 
aircraft  intercom  system  with  headsets. 

Flight  instruments  in  the  Frasca  are  arranged  in  the  standard  T  formation  in  front  of  the  pilot’s 
seat.  The  avionics  stack  is  directly  in  front  of  the  copilot’s  seat.  The  electronic  display  screen  is 
in  the  position  of  the  center  avionics  stack  in  a  Piper  Aztec.  The  electronic  display  screen 
measures  8.5-in  wide  by  6.5-in  high  with  multi-functional  buttons  along  the  bottom  and  right- 
hand  side  of  the  screen.  A  diagram  of  the  flight  instrument  panel,  including  the  electronic 
display,  is  presented  in  Appendix  A-2. 

The  Volpe  Center  developed  software  programs  to  simulate  flight  and  to  produce  the  maps 
presented  on  the  electronic  display  screen.  The  maps  were  shown  on  a  plan  view  display  in  the 
upper  left  hand  comer  of  the  screen,  measuring  approximately  4.5-in  wide  by  5-in  high. 


3.3  ELECTRONIC  DISPLAY  FORMATS 

The  electronic  display  formats  used  in  this  series  of  experiments  were  consistent  in  size,  color 
attributes,  and  symbology.  Terrain  was  displayed  in  one  of  two  ways:  as  green  contour  lines  of 
uniform  color  on  a  black  background  with  green  elevation  labels  on  the  lines  or  as  solid,  stacked 
polygons  in  a  range  of  greens — dark  greens  for  the  lowest  elevations  to  light  greens  or  yellow  for 
the  highest  elevations.  Elevation  labels  were  placed  within  the  solid  polygon. 

Both  terrain  presentations  used  similar  conventions  to  describe  elevation.  For  the  contour  lines, 
the  maximum  elevation,  above  mean  sea  level  (MSL),  for  the  area  between  two  bounding  lines, 
was  described  by  the  lower  of  the  two  elevation  labels.  For  the  solid  contour  shapes,  the 
maximum  elevation,  above  mean  sea  level,  for  an  area  of  a  certain  color,  was  described  by  the 
elevation  label  within  that  polygon. 

Navigation  information  on  the  electronic  display  screen  employed  formats  consistent  with 
previous  research  using  the  Frasca  simulator.'*  Purple  lines  indicated  airways.  Global 


*  Hannon,  D.J.,  and  Huntley,  M.S.,  Jr.  Electronic  Depiction  of  Instrument  Approach  Procedure  (lAP)  Charts.  Phase 
I:  Development  and  Preliminary  Evaluation.  Final  Report  DOT-VNTSC-FAA-95-14.  May  1995. 


Positioning  System  (GPS)  waypoints,  VORs,  and  intersections  were  presented  in  white  and 
labeled  with  the  appropriate  identifiers.  Runways  were  indicated  with  a  yellow  line  that  was 
thicker  than  the  airway  lines.  Labels  were  shown  in  capital  letters. 

The  airplane  shape,  shown  in  yellow,  was  the  same  size  in  all  experiments.  Information  on  the 
screen  moved  relative  to  the  airplane’s  course  changes.  For  north-up  oriented  maps,  the  airplane 
symbol  moved  along  the  map  until  it  reach  an  edge  of  the  screen.  The  map  would  update  and  the 
airplane  would  appear  in  the  middle  of  the  screen  with  corresponding  terrain  and  navigation 
information.  For  track-up  (inside-out)  oriented  maps,  the  airplane  symbol  remained  heading  up 
and,  in  a  constant  position,  near  the  bottom  of  the  screen,  and  centered  horizontally. 


3.4  PROTOCOL 

Procedural  protocols  were  reviewed  and  approved  by  the  MIT  Committee  on  the  Use  of  Human 
Subjects.  Accordingly,  pilots  signed  an  informed  consent  form  and  filled  out  an  information 
sheet  prior  to  research  and  were  debriefed  at  completion  of  the  study. 


4.  TERRAIN  DEPICTIONS  AND  NAVIGATIONAL  ACCURACY 


Electronic  displays  with  terrain  depictions  could  be  useful  for  navigation  when  there  is 
potentially  dangerous  terrain  and,  consequently,  a  requirement  for  a  high  degree  of  flight 
accuracy  on  approach.  The  purpose  of  the  first  experiment  was  to  determine  whether  electronic 
displays  of  terrain  affect  the  pilot’s  navigational  performance  during  instrument  approaches. 
Previous  research  has  suggested  that  providing  aircraft  position  information,  relative  to  the 
terrain,  increases  pilot  accuracy  when  compared  to  a  display  with  no  position  information.^ 
However,  if  the  pilot  does  not  recognize  that,  based  on  space  and  pixel  limitations  of  the  display, 
the  scale  of  terrain  depictions,  relative  to  other  symbols,  i.e.,  airplane  symbol,  is  disproportionate, 
a  pilot  might  tend  to  overcorrect  his  or  her  track  to  avoid  terrain. 

The  experimental  question  was,  is  pilots’  cross  track  error  (XTE)  more  likely  to  deviate  from  the 
flight  path  when  they  use  maps  with  terrain  features  as  compared  with  maps  without  terrain 
features?  Variation  in  XTE  would  indicate  that  pilots  deviated  from  their  intended  course  due  to 
terrain  representations  which  are  not  in  scale  with  the  aircraft  symbol.  (XTE  is  measured  as  the 
perpendicular  lateral  distance  of  the  airplane’s  actual  location  to  the  course  line.) 


4.1  PROCEDURE 

Participants  in  this  experiment  flew  three  fictional  GPS  approaches,  two  of  which  had 
corresponding  electronic  displays.  Appendix  B-1  shows  an  example  of  the  electronic  approach 
display  used  in  this  experiment.  The  participants  were  aided  by  maps  on  the  electronic  display 
screen  and  corresponding  paper  instrument  approach  charts  when  they  flew  these  approaches. 

The  paper  instrument  approach  charts  created  for  this  experiment  were  similar  in  format  and 
information  to  National  Oceanic  and  Atmospheric  Administration  (NOAA)  or  Jeppesen  approach 
charts  (Appendix  B-2  shows  a  paper  approach  chart  used  in  this  experiment). 

The  three  approaches  had  similar  geometry  (turn  angles,  rate  of  descent,  rate  of  ascent)  but  varied 
in  headings  and  elevations.  These  approaches  showed  mountain  peaks  at  four  relevant  positions: 
on  an  initial  approach  turn,  adjacent  to  the  final  approach  leg,  adjacent  to  the  missed  approach 
leg,  and  on  the  missed  approach  turn.  The  approach  used  as  a  baseline,  to  establish  flight 
technical  ability,  only  differed  in  that  it  did  not  present  terrain  information  or  use  an  electronic 
display. 

The  four  target  terrain  features  on  each  of  the  two  experimental  approach  were  positioned  close 
to  the  course  line  and  in  conformance  with  Terminal  Instrument  Procedures  (TERPS)  minimum 
clearances.  The  electronic  charts  were  drawn  such  that  terrain  contours  were  close  to,  but  did  not 
cross,  the  flight  path.  The  terrain  features  had  a  lower  elevation  than  the  closest  anticipated  point 


^  Amar,  M.J.,  Hansman,  Jr.,  R.J.,  Hannon,  D.J.,  Vaneck,  T.W.,  and  Chaudhry,  A.I.  A  Human  Subject  Evaluation  of 
Airport  Surface  Situational  Awareness  Using  Prototypical  Flight  Deck  Electronic  Taxi  Chart  Displays.  Final  Report 
DOT-VNTSC-FAA-95-36  (DOT/FAA/AR-95/I03).  November  1995. 


on  the  flight  path.  Position  of  the  terrain,  relative  to  the  course  line  (left  or  right),  varied  between 
the  two  approaches. 

Participants  flew  all  approaches  from  the  initial  approach  fix  to  the  end  of  the  published  missed 
approach  procedure  under  IFR  conditions  and  were  instructed  to  maintain  certain  aircraft 
performance  measures  and  to  perform  the  normal  activities  of  an  approach  procedure. 
(Participants  used  the  Flight  Checklist  shown  in  Appendix  B-3.)  Participants  were  given 
unlimited  practice  flight  time  to  become  familiar  with  the  simulator. 

When  the  participants  felt  comfortable,  they  flew  six  trials  (i.e.,  a  practice  approach,  two  baseline 
approaches,  and  three  experimental  approaches).  Table  4-1  shows  the  sequence  of  trials  as  well 
as  the  navigation  and  terrain  information  shown  on  the  electronic  display.  Participants’  XTE  was 
measured  for  all  approaches. 


Table  4-1.  Sequence  of  trial  presentation  for  approaches  by  information  on  the  electronic 

display. 


Trial 

Number 

Approach 

Approach 

Subject  1 

Subject  2 

Practice 

Experimental  Approach  1 

Experimental  Approach  2 

None 

1 

Baseline  Approach 

Baseline  Approach 

None 

2 

Experimental  Approach  2 

Experimental  Approach  1 

Terrain 

3 

Experimental  Approach  1 

Experimental  Approach  2 

Terrain  and  navigation 

4 

Experimental  Approach  2 

Experimental  Approach  1 

Terrain  and  navigation 

5 

Baseline  Approach 

Baseline  Approach 

None 

The  participants  were  debriefed  after  completing  six  trials  and  were  asked  for  their  reactions  to 
the  electronic  charts.  The  Post-Flight  Questionnaire  used  in  the  debriefing  is  in  Appendix  B-4. 


4.2  RESULTS 

The  mean  XTE  for  each  participant’s  baseline  approach  flights  was  within  requirements  for  flight 
competency.  The  maximum  baseline  XTE  occurrence  was  0.5  miles,  but  XTE  was  most  often 
less  than  0.1  miles.  For  the  experimental  approaches,  the  maximum  occurrence  of  XTE  was 
0.745  miles,  which  is  also  within  requirements  for  flight  competency.  Figure  4-1  shows  a  scatter 
plot  of  the  XTE  at  the  obstacle  points  (labeled  A,  B,  C,  and  D)  for  participants  flying  the 
experimental  approaches.  Participants  had  the  opportunity  to  use  an  electronic  moving  map 
display  when  flying  these  approaches. 
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Figure  4-1.  Flight  cross  track  error  (XTE)  at  each  obstacle  point  across  trials. 
(Obstacle  points  located  at  A,  B,  C,  and  D) 


The  XTE  at  the  obstacle  points  reveals  that  the  participants  were  as  likely  to  err  toward  the 
terrain  as  away  from  it  (24  incidents;  12  toward,  12  away).  The  XTE  is  evenly  distributed  about 
the  flight  path  with  no  skew.  In  debriefing,  participants  reported  in  debriefing  that  they  did  not 
find  the  terrain  information  on  the  display  useful  for  making  the  approaches.  The  debriefing 
results  corroborate  the  XTE  results.  The  participants  used  the  map  display  as  a  navigation  aid  for 
heading,  elevation,  and  distance  information,  or  as  a  reference  for  position  relative  to  course  line. 


4.3  DISCUSSION 

Use  of  terrain  depictions  on  electronic  displays  was  not  associated  with  flight  performance  in  this 
experiment.  Participants  did  not  alter  their  flight  path  systematically  when  the  electronic  display 
showed  their  airplane  symbol  flying  near  nonthreatening  terrain  depictions.  Since  the  safe  terrain 
did  not  negatively  impact  navigation,  further  research  is  warranted  to  examine  issues  related  to 
avoidance  of  dangerous  terrain. 


5.  DETECTING  AND  USING  POSITION  INFORMATION 


Before  advocating  the  use  of  electronic  displays  for  terrain  avoidance,  it  is  necessary  to  know 
how  readily  pilots  can  interpret  terrain  information  from  supplemental  electronic  displays.  In  this 
experiment,  participants  were  routed  off  their  original  airway  toward  terrain  that  was  at  a  higher 
elevation  than  the  aircraft.  The  experimental  scenario  made  it  difficult  for  participants  to  avoid 
terrain  impact  if  they  used  only  the  standard  flight  instruments  and  paper  charts.  The 
supplementary  electronic  display  presented  relative  position  information  that  made  it  easier  to 
detect  the  dangerous  terrain.  If  the  pilots  did  not  use  the  terrain  and  position  information  on  the 
electronic  display,  the  rerouting  in  the  scenario  would  cause  them  to  impact  terrain. 


5.1  PROCEDURE 

In  addition  to  holding  instrument  and  multi-engine  ratings,  more  than  half  of  the  twelve 
participants  held  Commercial  and  ATP  ratings  and  three  were  Certified  Flight  Instructors  (CFI). 
The  mean  age  of  the  eleven  men  and  one  woman  participants  was  52  years  (SD  =  17).  Table  5-1 
shows  participants’  flight  experience. 


Table  5-1.  Participants’  flight  experience. 


Total  Flight  Hours 
(in  hours) 

instrument  Flight  Hours 
(in  hours) 

Time  Since  Last  Flight 
(in  days) 

Range  =  586-17,000 
Mean  =  3,120 
(SD  =  5,290) 

Range  =  70-10,000 

Mean  =  1 ,625 
(SD  =  2,964) 

Median  =  6 

Participants  were  assigned  to  one  of  three  Map  Conditions  and  asked  to  carry  out  a  simulated 
flight  to  Albany,  NY.  Four  pilots  were  randomly  assigned  to  each  of  the  three  Map  Conditions. 
The  descriptive  statistics  for  the  participants  in  each  of  the  three  Map  Conditions  are  show  in 


Table  5-2. 


Table  5-2.  Participants’  age  and  flight  hours  by  Map  Condition. 


Map  Conditions 

Age 

(in  years) 

Total  Flight  Hours 
(in  hours) 

instrument  Flight  Hours 
(in  hours) 

Map  Condition  1 : 

No  Display, 

Paper  Chart  Only 

Mean  =  56 
(SD  =  13) 

Mean  =  7,550 
(SD  =  5,204) 

Mean  =  1,713 
(SD  =  2,209) 

Map  Condition  2: 

Display  with  Terrain  and 
Navigation  Information 

Mean  =  62 
(SD  =  8.8) 

Mean  =  5,810 
(SD  =  7,511) 

Mean  =  2,719 
(SD  =  4,856) 

Map  Condition  3: 

Display  with  Terrain 
Information  Only 

Mean  =  38 
(SD  =  20) 

Mean  =  2,559 
(SD  =  1,413) 

Mean  =  443 
(SD  =  526) 

The  Map  Condition  assignment  determined  the  type  of  electronic  map  display,  if  any,  provided  to 
the  participants  on  the  supplemental  display  screen.  Participants  in  Map  Condition  1  used  a 


paper  chart  and  were  not  provided  an  electronic  display.  Participants  in  Map  Condition  2  were 
given  a  paper  chart  and  an  electronic  display  with  terrain  and  navigation  information. 

Participants  in  Map  Condition  3  were  provided  with  a  paper  chart  and  an  electronic  display  of 
terrain  information  only. 

The  participants  in  Map  Conditions  2  and  3  used  a  Volpe-designed  electronic  moving  map  of  the 
area  between  Poughkeepsie  and  Albany  (approximately  40  miles)  which  had  1000  foot  terrain 
contour  lines,  navigation  information,  and  a  north-up  orientation.  The  navigation  information, 
only  available  to  participants  in  Map  Condition  2,  included  airways  labeled  with  Victor  airways 
numbers,  VORs,  and  intersections.  Appendices  C-1  and  C-2  show  the  displays  used  in  Map 
Conditions  2  and  3. 

Prior  to  flight,  the  researcher  reviewed  the  Frasca  instruments,  radio  stack,  and  the  electronic 
map  symbols  with  the  participant,  where  appropriate,  without  revealing  the  study’s  goal  of 
examining  terrain  avoidance  displays.  The  Frasca  simulator  generated  moderate  turbulence 
during  the  entire  experiment  to  increase  workload. 

First,  the  participant  flew  a  practice  flight  and  was  instructed  to  perform  normal  activities  of  en 
route  flight  while  becoming  familiar  with  the  simulator.  The  participant  used  a  NOAA  sectional 
chart  of  the  area  around  Allentown  for  navigation  and  dialed  up  the  Allentown  VOR  for 
familiarization.  The  researcher  answered  the  participants’  questions  over  the  headset. 

Participants  were  given  unlimited  practice  time,  but  most  spent  approximately  20  minutes. 

After  the  practice  flight,  but  prior  to  the  experimental  trial,  the  participant  completed  an  en  route 
paper  chart  familiarization  task,  which  required  him  or  her  to  identify  ten  items  circled  in  red  on 
a  photocopy  of  a  NOAA  sectional  chart  for  another  area.  All  participants  completed  this  task 
accurately. 

On  the  experimental  flight,  the  participants  were  instructed  to  fly  to  Albany.  They  used  a 
sectional  chart  of  the  area  around  Albany  and  approach  plates  for  both  Albany  and  Poughkeepsie. 
The  airplane  started  in  level  flight  at  3000  ft  above  sea  level,  which  was  the  Minimum  En  Route 
Altitude  (MEA)  on  the  originating  airway  (VI 23).  The  researcher,  as  Air  Traffic  Control  (ATC), 
communicated  with  the  pilot  using  the  headset  and  vectored  the  participant  around  weather  off 
the  original  airway  with  no  altitude  change.  Shortly  before  the  pilot  reached  a  new  airway,  V213, 
the  researcher  reported  loss  of  radar  contact  and  cleared  the  pilot  direct  to  Albany.  The 
participant  was  still  flying  at  an  altitude  of  3000  ft,  although  the  new  airway  had  a  Minimum 
Obstacle  Clearance  Altitude  (MOCA)  of  5500  ft.  The  chart  of  this  flight  path  is  presented  in 
Appendix  C-5  and  the  ATC  script  is  presented  in  Appendix  C-4. 

If  the  participant  noticed  the  terrain  and  questioned  the  current  altitude,  ATC  issued  a  higher 
clearance  and  the  trial  ended.  If  the  participant  did  not  question  ATC  immediately  after  radar 
contact  was  lost,  the  participant  flew  for  approximately  ten  more  minutes  to  allow  time  to  notice 
the  terrain  conflict,  albeit  too  late  to  take  corrective  action.  If  the  participant  still  did  not  notice 
the  higher  terrain,  the  trial  was  terminated. 


During  debriefing,  the  researchers  explained  that  the  experimental  goal  was  to  see  how  effective 
the  electronic  display  was  in  improving  pilot  ability  to  avoid  terrain.  The  researchers  emphasized 
that  the  study  was  not  a  test  of  piloting  skills,  but  was  intended  to  demonstrate  the  potential 
usefulness  of  the  display. 


5.2  RESULTS 

As  shown  in  table  5-3,  83%  of  the  participants  failed  to  detect  the  terrain  conflict.  Half  of  the 
participants  (two  out  of  four)  in  Map  Condition  3,  electronic  display  with  terrain  information 
only,  detected  the  higher  terrain.  Neither  participant,  however,  caught  the  error  in  time  to  avoid 
impact  with  the  terrain.  One  of  the  participants  in  Map  Condition  3  that  did  not  detect  the  terrain 
reported,  in  debriefing,  that  he  had  noticed  the  terrain  elevation  but  did  not,  and  would  not, 
question  ATC.  The  remaining  eight  participants  in  the  other  two  Map  Conditions  did  not 
recognize  the  terrain  conflict.  One  participant  avoided  the  terrain  due  to  tuning  the  wrong  VOR. 


Table  5-3.  Distribution  of  participants  by  whether  or  not  terrain  conflict  was  detected. 


Map  Conditions 

Detected 

Terrain  Conflict 

Did  Not  Detect 
Terrain  Conflict 

%  of  Total 

Map  Condition  1: 

No  Dispiay, 

Paper  Chart 

0 

4 

33% 

Map  Condition  2: 

Display  with  Terrain  and 
Navigation  Information 

0 

4 

33% 

Map  Condition  3: 

Display  with  Only  Terrain 
Information 

2 

2 

33% 

%  of  Total 

17% 

83% 

100% 

5.3  DISCUSSION 

These  results  suggest  that  the  participants  focused  on  terrain  elevation  when  it  was  highlighted  as 
in  Map  Condition  3,  where,  for  example,  the  participants  used  a  display  that  presented  only 
terrain  information.  This  interpretation  was  supported  by  the  comments  during  debriefing  from 
the  participants  who  detected  the  terrain  conflict.  These  participants  reported  that  they  detected 
the  higher  elevation  terrain  because  they  had  been  thinking  about  the  reasons  for  presenting  a 
terrain-only  electronic  display.  If  they  did  not  have  this  terrain-only  display,  these  participants 
reported  that  they  would  not  have  thought  about  terrain  during  the  scenario. 

Participants’  comments  supported  the  interpretation  that  pilots  do  not  tend  to  think  of  terrain  as 
an  issue  when  flying  en  route.  A  few  participants  mentioned  that  they  might  have  been  more 
alert  to  potentially  dangerous  terrain  if  they  had  been  flying  in  the  mountainous  western  United 
States.  One  third  of  the  participants  reported  that  they  depend  on  ATC  for  altitude  guidance  and 
were  trained  not  to  question  ATC. 


Kuchar  and  Hansman  also  demonstrated  this  high  level  of  trust  in  ATC.  In  their  experiment, 
pilots  were  vectored  into  terrain  by  ATC  and  had  a  very  low  initial  terrain  conflict  recognition 
rate,  regardless  of  the  electronic  display  presentation,  contours  or  spot  elevations.  However, 
once  the  pilots  recognized  that  ATC  might  not  give  accurate  vectors,  terrain  conflict  recognition 
increased  dramatically  when  the  pilot  was  provided  with  a  display  with  terrain  contour 
information.^ 

In  the  debriefing,  participants  mentioned  that  they  were  not  comfortable  trusting  the  electronic 
display,  which  was  unfamiliar,  when  it  contradicted  ATC  and  their  training.  Three  participants 
cited  the  high  workload  (i.e.,  the  turbulence  and  resulting  difficulty  in  maintaining  straight  and 
level  flight)  as  too  distracting  to  permit  them  to  include  the  display  into  their  scan.  One-third  of 
the  participants  said  that  if  they  had  bought  the  map  for  their  own  airplane,  understood  it  was  for 
terrain  alert,  and  had  practice  using  it,  they  would  have  noticed  the  terrain  conflict. 

Participants  presented  with  a  terrain  only-display  noticed  the  terrain  but  participants  with  the 
terrain  and  navigation  display  did  not.  This  suggests  that  the  additional  clutter  of  airways  may 
have  reduced  their  ability  to  focus  on  terrain.  However,  since  the  participants  in  Map  Condition 
1,  without  an  electronic  display,  failed  to  give  attention  to  terrain  issues,  the  underlying  issue 
may  be  attention.  These  results  proved  to  be  a  turning  point  in  the  conceptualization  of  this 
series  of  experiments.  The  crucial  issue  emerging  that  drawing  the  pilot’s  attention  to  the  display 
was,  possibly,  more  important  than  the  information  presentation.  The  responses  from  the 
participants  suggest  that  pilots  will  use  an  electronic  display  for  terrain  avoidance  if  their 
attention  is  drawn  to  the  display  for  a  terrain-related  reason,  and  are  presented  with  clear, 
informative  terrain  information. 


*  Kuchar,  J.K.  and  R.J.  Hansman  Jr.  (1991).  Advanced  terrain  displays  for  transport  category  aircraft  (Report  ASL- 
91-3).  Cambridge,  MA:  MIT  Aeronautical  Systems  Laboratory. 

Kuchar,  J.K.  and  R.J.  Hansman  Jr.  (1993).  An  exploratory  study  of  plan-view  terrain  displays  for  air  carrier 
operations.  International  Journal  of  Aviation  Psychology,  3(1),  39-54. 


6.  PREFERENCES  FOR  INFORMATION  DENSITY  ON  ELECTRONIC  DISPLAYS 

USED  FOR  TERRAIN  AVOIDANCE 


This  experiment  examined  pilots’  preferences  for  density  of  information  on  electronic  displays. 
Although  the  previous  experiment  suggested  that  minimal  information  on  an  electronic  display 
creates  the  clearest  presentation,  other  studies  have  indicated  that  pilots  prefer  maps  with  higher 
information  density.^  In  this  experiment,  participants  were  asked  what  map  features  would  have 
been  useful  to  detect  and  avoid  terrain  in  the  earlier  experiment.  The  participants  were  presented 
alternative  electronic  displays,  with  different  terrain  and  navigation  features.  Their  preferences 
were  recorded  to  see  if  they  agreed  with  the  previous  results. 


6.1  PROCEDURE 

The  eight  participants  had  participated  in  the  previous  experiment.  They  held  instrument  and 
multi-engine  ratings;  half  held  ATP  ratings,  and  two  were  also  CFIs.  The  average  participant  age 
(seven  men  and  one  woman)  was  55  years  (SD  =17  years).  Table  6-1  shows  participants’  flight 
experience. 


Table  6-1.  Participants’  flight  experience. 


Total  Flight  Hours 
(in  hours) 

Instrument  Flight  Hours 
(in  hours) 

Range  =  586-1 7,000 

Mean  =  5,372 
(SD  =  5,412) 

Range  =  102-10,000 

Mean  =  1 ,747 
(SD  =  3,354) 

Median  =  7.5 

Participants  viewed  twelve  alternative  electronic  display  presentations  on  a  desktop  computer 
and  on  the  electronic  display  screen  in  the  Frasca  242  simulator.  They  evaluated  twenty-four 
static  electronic  maps  each  of  which  had  a  standard  yellow  aircraft  symbol  in  the  center  for 
reference.  The  airways  were  labeled  with  MOCAs  rather  than  Victor  airway  numbers. 
Participants  rated  the  displays  using  paired  comparisons  and  individual  evaluations. 

Each  electronic  display  showed  three  types  of  information;  contour  type,  contour  increment  level, 
and  navigation  information  available.  Appendices  D-1  through  D-3  show  three  representative 
display  alternatives.  There  were  two  terrain  layouts  for  each  terrain  increment  level. 

Each  participant  was  shown  twelve  maps,  one  from  each  cell  in  the  design.  Table  6-2  has  twelve 
cells,  each  of  which  has  a  unique  combination  of  alternatives.  The  participants  viewed  all  the 
display  formats  on  a  desktop  computer  prior  to  evaluating  them.  Participants  were  instructed  to 
look  at  the  displays  and  develop  their  own  evaluation  criteria,  with  terrain  avoidance  as  the  main 
focus.  Participants  received  an  instruction  sheet  which  included  suggestions  for  rating  criteria 
(see  Appendix  D-4  for  a  copy  of  this  form).  Each  participant  was  asked  to  evaluate  how  useful 


’  Kuchar,  J.K.  and  R.J.  Hansman  Jr.  (1993).  Results  of  the  terrain  database  resolution  requirements  study  (Report 
ASL-93-4).  Cambridge,  MA:  MIT  Aeronautical  Systems  Laboratory. 


each  display  format  was  for  alerting  a  pilot  to  a  terrain  conflict  in  a  situation,  similar  to  the 
previous  experiment,  where  a  pilot  is  vectored  off  course  and  not  given  a  new  clearance. 


Table  6-2.  Electronic  display  map  formats. 

(Note:  two  terrain  layouts  were  created  for  each  terrain  increment  level.) 


!  CONTOUR  TYPE  I 

CONTOUR 

INCREMENT 

LEVEL 

I  SOLID  COLOR  CONTOURS 

I  CONTOUR  RINGS  I 

Naviaation  Information 

No  Naviaation  Information 

Naviaation  Information 

No  Naviaation  Information 

500’ 

Layouts  1  &  2 

Layouts  1  &  2 

Layouts  1  &  2 

Layouts  1  &  2 

1000’ 

Layouts  3  &  4 

Layouts  3  &  4 

Layouts  3  &  4 

Layouts  3  &  4 

BSSHi 

Layouts  5  &  6 

Layouts  5  &  6 

Layouts  5  &  6 

Layouts  5  &  6 

The  participants  rated  the  displays  using  the  paired  comparison  technique.^  Participants 
compared  each  of  the  twelve  alternative  formats  to  each  of  the  eleven  others,  for  a  total  of  66 
trials.  The  participants  were  asked  to  think  carefully  about  each  comparison  and  were  given 
unlimited  time  to  complete  the  task.  In  each  trial,  two  static  charts  were  presented  on  a  screen 
side  by  side  with  a  1 7-point  radio  button  rating  system  located  underneath  the  charts  at  the 
bottom  of  the  screen.  The  rating  system  ranged  from  “absolutely  better”  for  one  chart  to  “same” 
in  the  center  (between  the  two)  to  “absolutely  better”  for  the  other  chart  (see  Appendix  D-5  for 
an  image  of  the  paired  comparison  program).  Participants  were  instructed  to  select  the  radio 
button  on  the  rating  scale  that  corresponded  with  their  rating  for  that  pair  of  charts.  They  could 
change  their  responses  at  any  time,  including  responses  to  previous  trials. 

For  the  paired  comparisons,  the  twelve  design  alternatives  received  scores  based  on  the 
participant’s  selection.  The  scores  ranged  from  1  to  17,  where  1  indicated  a  rating  of  most 
preferred,  9  indicated  a  rating  of  equal  preference  and  17  indicated  a  rating  of  least  preferred. 
When  a  participant  indicated  a  preference  for  one  chart,  that  chart  was  given  a  low  rating, 
between  1  and  9,  that  corresponded  with  the  gradation  of  preference.  The  nonpreferred  chart  was 
assigned  a  rating  of  18,  minus  the  preferred  chart’s  rating. 

The  participants  then  viewed,  and  evaluated,  each  display  format  using  the  electronic  display  in 
the  Frasca  simulator.  The  Frasca  simulator  display  gave  the  participants  a  more  realistic  context 
for  evaluating  the  charts’  function.  The  participants  evaluated  each  format  using  a  display  rating 
questionnaire  (shown  in  Appendix  D-6).  Following  this,  participants  were  debriefed  on  their 
opinions  and  suggestions. 


6.2  RESULTS 

6.2.1  Paired  Comparison  Task 

The  mean  rating  scores  for  the  two  terrain  layouts  within  every  cell  of  the  design  were  analyzed 
using  Wilcoxon  Rank  Sums.  Since  there  were  no  significant  differences  between  the  ratings  for 


This  technique  was  based  on  the  one  used  in  Amar,  M.J.,  et  al.  A  Human  Subject  Evaluation  of  Airport  Surface 
Situational  Awareness  Using  Prototypical  Flight  Deck  Electronic  Taxi  Chart  Displays.  Final  Report  DOT-VNTSC- 
FAA-95-36  (DOT/FAA/AR-95/103).  November  1995. 


terrain  layout,  the  rating  scores  for  terrain  layouts  in  each  cell  were  pooled  and  descriptive 
statistics  calculated.  These  results  are  presented  in  table  6-3. 


Table  6-3.  Paired  comparison  ratings  for  each  display  format. 
(1  =  most  preferred  format,  17  =  least  preferred  format). 


Solid  color 
contours  and 
Navigation 
information 

Solid  color 
contours  and 
no  Navigation 
information 

Contour  rings 
and  Navigation 
information 

Contour  rings 
and  no  Navigation 
information 

500’ 

increment 

level 

Mean  =  3.65 
(SD  =  3.00) 

Mean  =  8.28 
(SD  =  4.68) 

Mean  =  7.24 
(SD  =  4.46) 

Mean  =  11 .05 
(SD  =  4.13) 

1000’ 

increment 

level 

Mean  =  5.84 
(SD  =  4.05) 

Mean  =  9.88 
(SD  =  4.59) 

Mean  =  8.20 
(SD  =  4.52) 

Mean  =  12.25 
(SD  =  3.40) 

2000’ 

increment 

level 

Mean  =  7.34 
{SD  =  4.31) 

Mean  =  1 1 .58 
(SD  =  4.04) 

Mean  =  9.35 
(SD  =  4.63) 

Mean  =  13.34 
(SD  =  2.88) 

Table  6-3  shows  that  the  most  preferred  display  format  (rating  =  3.65,  SD  =  3.00)  had  solid  color 
contours,  navigation  information,  and  500’  contour  increments.  The  least  preferred  display 
(rating  =  13.34,  SD  =  2.88)  had  contour  rings,  no  navigation  information,  and  2000’  contour 
increments. 

The  graph  in  figure  6-1  shows  that  participants  preferred  more  detail,  i.e.,  navigation 
information,  finer  grained  contour  increments,  and  solid  color  contours. 


Contour  Rngs  -  No 
Navigation  Information 


Solid  Color  -  No 
Navigation  Information 


Contour  Rings  - 
Navigation  Information 


Solid  Color  -  Navigation 
Information 


Figure  6-1.  Preferences  by  contour  increment  level. 
(Note;  rating  score  inversely  related  to  preference.) 


6.2.2  Individual  Evaluation  Task 

Evaluation  of  alternative  display  formats  showed  that  participants  preferred  a  higher  density  of 
information,  as  indicated  in  the  paired  comparison  results.  Generally,  they  preferred  finer 
grained  contour  increment  levels  (500  ft  or  1000  ft).  Most  of  the  participants  preferred  the 
denser  information  alternative,  especially  the  positional  orientation  provided  by  navigation  and 
terrain  information,  and  the  solid  color  contours. 

This  preference  for  solid  color  contours  reflected  participants’  preference  for  color  to  change 
relative  to  elevation,  which  allows  graphical  information  to  substitute  for  digital.  Participants 
also  reported  that  the  solid  color  contours  gave  them  a  better  three  dimensional  perception  of 
elevation.  Only  two  participants  said  they  disliked  the  solid  color  contours  because  the  colors 
cluttered  the  display.  Three  participants  suggested  that  unfilled  contours  could  create  visual 
space  for  more  and  different  information. 

Participants  suggested  the  following  ways  to  improve  the  displays: 

•  Reduce  clutter  by  providing  fewer  elevation  labels  and  allowing  color  gradation  to  describe 
the  curves. 

•  Place  an  altitude  tag  on  the  airplane  to  provide  a  clearer  sense  of  the  relationship  between  the 
airplane  and  the  terrain. 

•  Restrict  navigation  information  to  the  current  flight  path.  (It  is  not  necessary  to  include  all 
the  airways.) 

•  Include  more  airway  information  such  as  VOR  frequencies,  DMEs,  and  assigned  altitudes. 

•  Eliminate  en  route  MOCAs  when  terrain  contours  and  elevation  labels  are  present.  MOCAs 
should  be  replaced  by  airway  numbers  (this  was  the  format  in  the  previous  experiment). 

•  Add  range  information. 

•  Add  distance  and  heading  to  nearest  airport. 

•  Indicate  terrain  higher  than  the  airplane  in  red. 

•  Use  track-up  display  presentations. 

63  DISCUSSION 

Participants  preferred  to  use  electronic  displays  with  dense  information  to  improve  their  terrain 
awareness.  They  preferred  graphical  presentations  to  reduce  redundancy  and  saw  graphical 
presentations  as  freeing  space  for  additional  information.  These  results  are  consistent  with 
previous  research.^  However,  the  previous  experiment  indicated  that  a  high  density  of 
information  on  the  displays  was  not  the  best  way  to  foster  positional  and  terrain  awareness.  It 
appears  that  how  the  information  was  presented  was  secondary  to  getting  the  pilots  to  look  at  the 
display.  Unless  the  display  captures  the  pilot’s  attention,  it  cannot  be  useful  for  terrain 
avoidance.  With  these  results  in  mind,  the  final  experiment  was  designed  to  explore  the  usability 
of  display  formats  paired  with  visual  alerts. 


’  Kuchar,  J.K.  and  R.J.  Hansman  Jr.  (1993).  Results  of  the  terrain  database  resolution  requirements  study  (Report 
ASL-93-4).  Cambridge,  MA;  MIT  Aeronautical  Systems  Laboratory. 


7.  VISUAL  ALERTS  AND  THE  INTERPRETABILITY  OF 
ELECTRONIC  DISPLAYS  OF  TERRAIN 


Since  the  display  is  not  in  the  primary  field  of  view,  the  three  prior  experiments  explored  how 
pilots  incorporate  into  their  instrument  scan  a  supplemental  electronic  display  which  did  not  have 
warning  cues.  This  final  experiment  measured  the  effectiveness  of  a  visual  alert  presented 
together  with  an  electronic  display  for  terrain  avoidance.  If  the  visual  alert  presented  in  this 
experiment  highlights  potential  problems,  this  alert  could  help  pilots  to  better  interpret  a  range  of 
display  formats,  including  those  with  dense  information. 


7.1  PROCEDURE 

Six  instmment-  and  multi-engine-rated  pilots  participated  in  this  experiment;  four  of  the  six  held 
Commercial  and  ATP  ratings,  and  four  held  CFI  ratings.  All  the  participants  in  this  test  were 
male  with  an  average  age  of  42.5  years  and  a  standard  deviation  (SD)  of  14  years.  Table  7-1 
shows  participants’  flight  experience. 


Table  7-1.  Participants’  flight  experience. 


Total  Flight  Hours 
(in  hours) 

instrument  Flight  Hours 
(in  hours) 

Range  =  593-28,000 

Mean  =  5,740 
(SD  =  10,922) 

Range  =  60-2,800 

Mean  =  779 
(SD=  1,096) 

Median  =  8.5 

The  participants  flew  the  Frasca  242  simulator  and  were  required  to  react  to  the  display  screen  by 
pressing  one  of  the  six  buttons  located  below  the  screen.  There  was  a  clearly  marked  label  above 
each  button.  The  labels,  from  left  to  right,  were,  “No  Action,”  “Left,”  “Climbing  Left,”  “Climb,” 
“Climbing  Right,”  and  “Right”. 

The  electronic  display  formats  varied  on  three  dimensions:  map  density  (high,  medium,  and 
low),  airplane  orientation  (track-up  and  north-up),  and  alert  type  (true  alarm  and  false  alarm). 

One  thousand  foot  contour  increments  were  present  on  all  the  maps.  The  high  density  map  had 
navigation  information  (airways,  VORTACs,  and  intersections)  and  solid  contour  shapes.  The 
medium  density  map  had  solid  contour  shapes  but  no  navigation  information.  The  low  density 
map  had  contour  rings  and  also  lacked  navigation  information.  Appendices  E-1  through  E-3 
show  examples  of  the  three  display  formats. 

Each  participant  flew  two  practice  flights  and  four  experimental  flights.  The  practice  flight  was 
25  miles  long  and  included  procedures  similar  to  the  experimental  runs,  but  only  presented  six 
alerts,  one  for  each  electronic  display  in  each  of  the  two  airplane  orientations;  north-up  and 
track-up.  The  experimental  flights,  100  miles  in  length  and  40  to  45  minutes  long,  were  identical 
to  one  another  for  navigational  purposes.  The  four  experimental  flights  differed  in  terrain  layout 
and  sequence  of  alerts  (i.e.,  independent  variables  manipulated).  Alerts  were  presented  twelve 
times  on  each  experimental  flight. 


The  researcher  briefed  each  participant  at  a  computer  terminal  prior  to  the  experiment.  The 
researcher  showed  the  participant  six  electronic  displays,  each  of  the  six  formats  presented,  and 
explained  how  to  interpret  the  electronic  displays.  The  participant  was  given  the  paper  chart  (see 
Appendix  E-4  for  a  copy  of  the  chart  with  the  routes  flown)  and  told  to  fly  the  assigned  route  at 
the  assigned  altitude.  The  paper  chart  included  information  present  on  standard  charts  such  as 
headings,  VORTACs,  and  airways,  but  did  not  provide  terrain  elevation  information.  The 
participant  was  told  to  respond  to  ATC  commands  (as  issued  by  the  researcher)  such  as  changes 
in  altimeter  setting,  communication  frequencies,  and  transponder  squawk  codes,  and  to  react  to 
the  alert  maps  as  accurately  and  quickly  as  possible.  (See  Appendix  E-5  for  a  flight  script).  The 
participant  was  informed  that  the  assigned  route  was  within  restricted  airspace  and  he  or  she 
should  not  deviate  from  course  unless  necessary. 

The  first  practice  flight  familiarized  the  participants  with  the  flight  dynamics  and  the  moderately 
high  turbulence  present  throughout  the  experiment  without  having  to  use  the  electronic  display. 
There  was  no  time  limit  set  for  this  practice  flight  but  most  participants  flew  for  approximately 
ten  to  fifteen  minutes. 

The  participant  flew  a  second  practice  flight  using  an  assigned  route  that  included  alert  triggers  to 
become  familiar  with  the  electronic  display  and  to  practice  responding  to  alerts.  During  this 
flight  and  the  subsequent  experimental  flights,  the  participant  was  required  to  maintain  5,000  feet 
and  170  to  180  knots  airspeed.  The  electronic  display  was  blank  and  navigation  was  by  VORs. 
The  participant  was  not  permitted  to  use  the  autopilot. 

During  the  flights,  the  researcher  would  trigger  an  alert  every  two  to  five  minutes,  presenting  the 
participant  with  a  potential  terrain  conflict.  During  an  alert,  the  airplane  would  continue  flying, 
the  screen  would  flash  twice,  displaying  a  red  background  with  the  word  “alert”  in  black  letters, 
and  then  the  map  display  would  appear.  The  participant  was  expected  to  assess  the  situation  and 
determine  what  action,  if  any,  was  necessary  to  maintain  terrain  separation.  The  participant  was 
instructed  to  keep  the  airplane  straight  and  level  and  to  respond  to  the  alert  by  pressing  a  button 
located  below  the  screen.  Pressing  a  button  blanked  the  screen.  The  participant  could  not  change 
his  mind  after  making  a  selection.  Reaction  time  (in  seconds)  was  measured  from  the  time  an 
alert  appeared  on  the  screen  to  the  time  the  pilot  pressed  a  button  (made  a  decision).  Percentage 
correct  responses  at  each  alert  point  were  recorded  for  every  flight  in  addition  to  flight  data  such 
as  plane  orientation,  speed  and  pitch. 

After  the  participant  completed  four  experimental  flights,  he  or  she  completed  a  questionnaire 
(see  Appendix  E-6)  to  obtain  subjective  responses  to  the  experiment.  The  researcher  used  the 
questionnaire  as  the  starting  point  to  discuss  the  session  with  the  participant. 


7.2  RESULTS 


The  mean  overall  reaction  time  to  alerts  was  5.5  seconds  with  a  standard  deviation  of  4.18 
seconds.  The  median  reaction  time  was  4.6  seconds.  (See  table  7-2  for  the  mean  reaction  times 
for  each  display  format  combination.)  There  were  no  significant  differences  in  reaction  time  by 
map  density,  airplane  orientation,  or  alert  type  using  factorial  Analysis  of  Variance  (ANOVA). 


Table  7-2.  Mean  reaction  times  to  alerts  for  display  formats. 


1  ALERT  TYPE  1 

AIRPLANE 
ORIENTATION  > 

1  TRUE  ALARM  1 

FALSE  Al 

ARM 

Track-UD 

North-uD 

Track-UD 

North-uD 

MAP 

DENSITY 

Low 

5.0  sec.,  3.2  sd 

4.8  sec..  1.5  sd 

6.2  sec..  7.5  sd 

5.2  sec..  2.1  sd 

Medium 

4.9  sec..  4.2  sd 

5.9  sec..  4.1  sd 

7.1  sec..  8.8  sd 

5.2  sec..  2.4  sd 

Hiah 

5.1  sec..  1 .8  sd 

6.3  sec.,  3.0  sd 

5.4  sec.,  2.4  sd 

5.2  sec.,  1.8  sd 

The  percentage  of  correct  responses  was  81%.  A  correct  response  was  defined  as  responding 
with  any  of  the  five  action  types  when  an  action  was  required  or  with  “No  Action”  when  no 
action  was  required.  Error  was  divided  into  two  categories:  “False  Positive”  errors  resulted  from 
participants  responding  with  one  of  the  five  types  of  actions  if  no  action  was  required,  and  “False 
Negative”  errors  which  resulted  from  the  participants  responding  “No  Action”  when  there  was  an 
action  required.  No  determination  was  made  in  advance  as  to  what  type  of  action  was  the  most 
appropriate  for  each  “action”  trial,  and  consequently,  type  of  action  was  not  analyzed  for 
accuracy.  Table  7-3  shows  accuracy  of  responses  to  alerts. 


Table  7-3.  Distribution  of  responses  to  alerts  by  correct  and  actual  (pilot)  response. 


Correct  Resoonse: 

Action 

Correct  Resoonse: 

No  Action 

Total  ResDonses 

Actual  ResDonse: 
Action 

43.1% 

11.8%* 

54.9% 

Actual  ResDonse: 

No  Action 

6.9%** 

38.2% 

45.1% 

Total  ResDonses 

50% 

50% 

100% 

*  These  errors  are  referred  to  as  False  Positive  because  they  occurred  when  no  action  was  necessary  and  the 
participant  responded  with  an  action. 

**  These  errors  are  referred  to  as  False  Negative  because  they  occurred  when  an  action  was  necessary  and  the 
participant  responded  with  no  action. 

The  nearly  19%  error  rate  led  to  the  question:  Were  there  alert  points  (trials)  along  the  flight  path 
that  produced  a  significantly  higher  number  of  errors  than  other  alert  points?  Error  rate  was 
examined  at  each  alert  point.  Percentages  of  correct  and  incorrect  responses  across  trials  are 
shown  in  table  7-4.  Four  points  along  the  flight  path  had  a  total  error  rate  of  25%  or  higher  (trials 
3,7, 10,  and  11). 


Table  7-4.  Percentages  of  correct  and  incorrect  responses  across  alert  points. 
(Bold  type  indicates  the  four  points  where  the  error  was  greater  than  25%  of  the  responses.) 


r  ALERT  POINTS  1 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

Correct  Responses 

95.8% 

75.0% 

91.7% 

87.5% 

83.3% 

62.5% 

87.5% 

83.3% 

70.8% 

66.6% 

00 

cn 

4.2% 

16.7% 

8.3% 

12.5% 

16.7% 

37.5% 

12.5% 

16.7% 

29.2% 

33,4% 

12.5% 

4.2% 

8.3% 

0.0% 

12.5% 

16.7% 

20.8% 

8.3% 

4.2% 

8.3% 

29.2% 

8.3% 

0.0% 

8.3% 

4.2% 

8.3% 

0.0% 

0.0% 

16.7% 

4.2% 

20.8% 

4.2% 

The  higher  incorrect  response  rate  at  some  alert  points  was  of  concern.  Perhaps  participants  did 
not  respond  to  the  alerts  in  the  manner  that  was  intended  at  these  points  or  participants  may  not 
have  been  oriented  appropriately  when  the  alert  was  triggered.  Using  a  program  to  examine  map 
type  and  airplane  position  and  heading  during  each  alert  trigger,  incorrect  responses  (false 
positives  and  false  negatives)  were  subcategorized.  Each  incorrect  response,  whether  it  was  a 
false  positive  or  a  false  negative,  was  classified  as  either  a  “real  error,”  or  as  one  four  t)^es  of 
“invalid  response,”  outlined  in  table  7-5. 


Table  7-5.  Invalid  response  categories  used  to  code  incorrect  responses. 

INVALID  RESPONSE  CATEGORIES 

•  Participant  responded  based  on  an  anticipated  navigational  maneuver  instead  of  based  on 

current  heading. _ _ _ 

•  Participant  was  provided  with  more  look  ahead  than  was  intended  and  the  participant 

responded  to  higher  terrain  ahead  that  was  not  supposed  to  be  seen.  (This  happened 
primarily  on  track-up  maps  because  the  airplane  was  often  placed  at  the  bottom  of  the  screen 
with  maximum  look  ahead  instead  of  in  the  center  on  north-up  maps.) _ 

•  Participant  was  not  on  the  expected  flight  path  when  the  alert  was  triggered  and  was 

therefore  oriented  incorrectly  to  the  terrain  features. _ 

•  Computer  glitches  prevented  the  researcher  from  triggering  the  alert  at  the  appropriate  time 
and  therefore  the  participant  was  oriented  incorrectly  to  the  terrain  features  (this  was  rare). 


Half  of  the  incorrect  responses,  19%  of  the  total  responses,  were  coded  as  invalid  responses.  A 
new  data  set  was  created  for  additional  analyses  excluding  the  invalid  responses,  approximately 
9%  of  the  data  points.  An  ANOVA  for  reaction  time  and  two  chi-squares  were  performed  on  the 
reduced  data  set;  these  results  were  not  significant. 


7.3  DISCUSSION 

There  were  no  significant  differences  in  interpretability  for  any  of  the  displays.  If  terrain 
elevations  were  clearly  marked,  pilots  were  able  to  quickly  interpret  displays  with  varying 
information  densities.  These  results  open  a  range  of  electronic  display  possibilities.  However,  if 
the  goal  of  this  type  of  research  is  to  create  a  display  that  produces  little  to  no  error,  then  error 
rate  in  this  experiment  indicated  that  the  format  presented  was  not  ideal.  Further  studies  to 
assess  generalizability  of  these  results  and  explore  ways  to  increase  accurate  interpretability  are 
needed. 


8.  OVERVIEW 


Terrain  depictions  on  maps  can  be  useful,  but  the  depictions  must  present  information  in  a  clear 
manner.  It  is  apparent  that  there  needs  to  be  a  mechanism  to  draw  the  pilot’s  attention  to  the 
display.  This  could  be  a  minimal  display  (such  as  the  terrain-only  map  in  chapter  5)  or  an 
alerting  system  used  in  conjunction  with  the  display  (visual  alert  in  chapter  7). 

This  research  also  supports  previous  work  that  indicates  pilots  prefer  displays  with  higher 
densities  of  information.  Electronic  displays  with  terrain  depictions,  of  varying  information 
densities,  combined  with  a  terrain  alerting  system,  will  allow  pilots  to  quickly  and  accurately 
interpret  information  useful  for  terrain  avoidance. 

The  major  issues  yet  to  be  addressed  which  were  raised  by  this  series  of  experiments  were: 
familiarity  with  electronic  displays  (amount  of  training),  clarity  and  interpretability  of  the  display 
(placement  and  presentation  of  elevation  information),  and  airplane  relative  position  on  the  map. 
Each  of  these  issues  need  to  be  addressed.  To  further  investigate  the  human  factors  aspects  of 
electronic  displays,  it  is  necessary  to  modify  training  practices  in  experimentation,  including 
training  to  proficiency.  One  possibility  for  modifying  elevation  information  is  to  label  elevations 
of  a  safety  floor  and  only  present  the  terrain  that  is  relevant  to  the  pilot  (terrain  at  his  altitude  or 
higher).  The  position  of  the  airplane  could  be  modified  to  provide  maximum  look  ahead,  instead 
of  in  the  middle  of  the  screen,  as  it  was  in  the  last  experiment  (chapter  7). 

Overall,  this  series  of  experiments  addressed  important  issues  for  designing  electronic  display 
terrain  avoidance  systems  and  of  interest  for  future  research. 
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Appendix  A>2.  Simulator  Flight  Instrument  Panel 
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Appendix  B-1.  Electronic  Approach  Display 
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Appendix  B-2.  Paper  Approach  Chart 
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Appendix  B-3.  Flight  Checklist 


Approach  number: 
Q  i  2  3 

Called  in  Final  Approach  _  _  _  _ 

Applied  carb  heat  (on) 

Boost  pumps  (on)  _  _  _  _ 

Wing  Flaps  (down)  _  _  _  _ 

Landing  gear  (down)  _  _  _  _ 

Called  in  Missed  Approach  _  _  _  _ 

Applied  carb  heat  (off)  _  _  _  _ 

Boost  pumps  (off)  _  _  _  _ 

Wing  Flaps  (up)  _  _  _  _ 

Cowl  Flaps  (open)  _  _  _  _ 

Landing  gear  (up)  _  _  _  _ 

Maintained  speed:  +/-  5  kts 
Maintained  altitude:  +/- 100  ft. 

Responded  to  ATC  when  flight 
corrections  requested 


Appendix  B-4.  Post-Flight  Questionnaire 


1.  What  kind  of  information  was  most  helpful  to  you  on  the  electronic  map? 


2.  In  particular,  was  the  terrain  depiction  useful  information? 

3.  Were  there  specific  legs  of  the  flight  where  the  electronic  map  facilitated  flying? 


4.  At  Waypoint  turns,  did  you  fly  through  them  or  anticipate  them? 

In  either  case,  did  you  use  the  moving  map  or  the  needle  for  the  turn? 


5.  Which  information  on  the  map  did  you  find  unnecessary? 


6.  Would  you  find  this  map  useful  in  VFR  conditions  as  well? 


7.  Any  other  suggestions  or  comments  on  the  map  display? 


8.  Any  other  suggestions  or  comments  on  the  flight  testing  procedure? 


Appendix  C-1.  Electronic  Display  for  Map  Condition  2:  Navigation  and  Terrain 

Information 


Appendix  C-2.  Electronic  Display  for  Map  Condition  3:  Terrain  Information 


Appendix  C-4.  Script  for  Experiment  2  (Albany  Experiment) 


Pilot  Briefing:  Airplane  starts  at  TRESA  at  3000’  on  V123  direct  to  Albany. 

Pilot  is  talking  to  NY  approach  control. 

AXIS  at  Albany  is  not  functioning. 

Autopilot  is  NOT  to  be  used. 

FUght: 

ATC:  “N123SH  This  is  New  York  approach.  Maintain  3000” 

(after  a  few  minutes  of  flight) 

ATC:  “N123SH  contact  Albany  approach  control  on  122.4” 

(pilot  contacts  Albany) 

Alb.  ATC:  “N123SH  Albany  altimeter  setting  is  two-niner-niner-seven” 

(at  WIGAN) 

ATC:  “N123SH  this  is  a  vector  around  weather  along  Victor  123.  Turn  Left  to  heading 
of  two-niner-zero  290.  Expect  15  miles.” 

(after  new  heading  is  established) 

ATC:  “N123SH  contact  Albany  on  125.025” 

(pilot  notices  error) 

ATC:  “Roger,  N123SH,  make  that  122.6” 

ATC:  “N123SH  your  transponder  is  intermittent,  squawk  code  2526  and  ident” 

(before  intersection  with  V213  and  terrain) 

ATC:  “N123SH,  radar  contact  has  been  lost,  you  are  cleared  direct  to  Albany” 

(if  pilot  questions  altitude) 

ATC:  “Roger,  expect  higher  in  2  miles” 


ATC:  “N123SH  climb  and  maintain  7000.” 


Appendix  D-1.  Static  Electronic  Display  Format:  500-ft  Contours,  Filled  Contours,  and 

Navigation  Information  • 
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ASTAR 

A 


STARK 

A 
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Appendix  D-2.  Static  Electronic  Display  Format:  1000-ft  Contours,  Filled  Contours,  and 

Navigation  Information 


Appendix  D-3.  Static  Electronic  Display  Form 

and  Navigation  I 


fill  Hi 


:  2000-ft  Contours,  Unfilled  Contours, 


Appendix  D-4.  Electronic  Display  Format  Preference  Task  Rating  Criteria 


For  this  part  of  the  study  we  will  ask  you  to  look  at  a  series  of  two  chart  comparisons.  There  will 
be  66  comparisons  in  all. 

Your  task  will  be  to  rate  the  maps  based  on  the  scenario  you  just  flew. 

Ask  yourself  the  following  question: 

Which  of  the  two  maps  would  have  been  more  useful  to  me  during  the  flight  to 
notice  and  avoid  terrain? 

Before  you  begin,  you  will  have  a  chance  to  look  at  each  map  individually  at  your  own  pace. 
During  this  time,  think  about  criteria  you  will  use  for  evaluation.  Please  try  to  maintain 
consistent  criteria  throughout  the  rating  task. 

The  following  are  a  few  suggestions  for  setting  criteria.  Please  note  they  are  only  suggestions: 

You  are  not  limited  to  these  nor  are  you  required  to  use  them. 

-  Is  the  level  of  clutter  useful? 

-  Is  the  level  of  detail  useful? 

-  Is  the  text  easy  to  interpret? 

-  Is  the  terrain  presentation  clear? 

-  Is  the  navigation  information  clear  and/or  necessary? 


You  will  be  able  to  pace  yourself  through  the  comparisons.  Click  on  the  rating  circle  that  most 
appropriately  matches  your  evaluation  of  the  maps.  Choosing  a  rating  will  not  advance  the 
computer  to  the  next  comparison.  Only  when  you  click  on  “Next”  will  the  computer  advance. 
You  may  also  go  back  to  previous  comparisons  if  you  want  to  change  your  mind. 


Appendix  D-5.  Static  Electronic  Display  Format  Comparisons 
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Appendix  D-6.  Display  Rating  Questionnaire 

Can  the  map  be  easily  interpreted?  Explain. 

What  suggestions  for  improvement  do  you  have? 

A.  What  information  should  or  shouldn’t  be  included? 

B.  What  are  the  best  and  worst  features  of  the  map? 

C.  In  what  other  ways  could  information  be  presented? 

Is  the  terrain  depiction  clear  and  easy  to  understand?  Explain. 

A.  Does  it  have  enough  detail?  If  no,  what  else  should  be  added? 

B.  Does  it  have  unnecessary  detail?  If  yes,  what  should  be  removed? 

If  the  map  contains  navigation  information: 

A.  Is  the  depiction  clear  and  easy  to  understand?  If  no,  why? 

B.  Is  there  enough  information?  Should  anything  be  added  or  removed? 


Appendix  E-1.  Low-Density  Display  Format 


Appendix  E-2.  Medium-Density  Display  Format 


Appendix  E-3.  High-Density  Display  Format 


BEEST 


Appendix  E-4.  Paper  Chart  Including  Flight  Paths  for  the 
Practice  and  Experimental  Runs 

practice  flight  indicated  with  dotted  line  and  experimental  flight  indicated  with  straight  line 
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Appendix  E-5.  Example  of  Flight  Script  (for  Experiment  4) 


Talking  to  ATC  at  Foreman  Airport  on  122.4 

ATC:  “N123SH  this  is  Foreman  Approach.  Maintain  5000  feet” 

1  (phase  1:  first  alert:  alarm) 

2  (phase  1:  second  alert:  alarm) 

3  (phase  1:  third  alert:  false) 

ATC:  “N123SH  altimeter  is  two-niner-niner-four” 

4  (phase  2:  first  alert:  alarm) 

ATC:  “N123SH  transponder  is  intermittent,  squawk  code  4021  and  ident” 

5  (phase  2:  second  alert:  false) 

ATC:  “N123SH  contact  Foreman  on  131.25” 

6  (phase  2:  third  alert:  alarm) 

ATC:  *m23SH Report  crossing  AYESr* 

7  (phase  3:  first  alert:  alarm) 

ATC:  *‘N123SH  contact  Foreman  on  122.75” 

ATC:  “N123SH  altimeter  is  29.99” 

8  (phase  3:  second  alert:  false) 

ATC:  “N123SH  transponder  is  intermittent  squawk  code  2040  and  ident” 

9  (phase  3:  third  alert:  false) 

ATC:  “N123SH  contact  Foreman  on  130.5” 

ATC:  “N123SH  transponder  is  intermittent,  squawk  code  3231  and  ident” 

10  (phase  4:  first  alert:  false) 

ATC:  “N123SH  altimeter  is  29.98” 

1 1  (phase  4:  second  alert:  false) 

ATC:  “N123SH  contact  Foreman  approach  on  128.75” 

12  (phase  4:  third  alert:  alarm) 


Appendix  E-6.  Post-Experiment  Questionnaire 


1.  Which  map(s)  did  you  find  the  easiest  to  interpret?  Why? 


2.  Which  map(s)  did  you  find  the  most  difficult  to  interpret?  Why? 


3.  In  what  other  ways  could  this  alert  information  be  presented? 


4.  Should  more  detail  be  added  to  any  of  the  maps?  If  yes,  what  else  should  be  added? 


5.  Did  any  of  the  maps  have  unnecessary  detail?  If  yes,  which  maps  and  what  should  be 
removed? 


6.  Which  orientation  of  the  map  (north  up  or  track  up)  did  you  prefer?  Why? 


7.  Any  other  suggestions/comments: 


8.  Have  you  had  any  flight  experience  in  mountainous  areas?  If  so,  how  much? 


